Following the recent PCA upgrade from 3.0.2-M3.5 to 3.0.2-M3.7.1, we encountered an issue where Grafana was unable to send alerts as intended. Since Grafana plays a crucial role in monitoring aspects such as the health state of ZFSSA, restoring this functionality became important. Upon raising a service request (SR), it was identified as a bug that is planned for resolution in version 3.0.2-M3.9, according to support. Therefore, if version 3.0.2-M3.8 is released in the future, we’ll need to verifay the situation again.
Previously, there was a MOS note detailing the configuration of SMTP for the earlier software release, identified by Doc ID 2964429.1. Unfortunately, this document is currently unavailable. I am hopeful that it will be updated in the near future.
Grafana SMTP is not enabled. Please contact #sauron-support for help
After applying the patch, attempting to send a test alert via the Grafana web interface resulted in the following error: “Grafana SMTP is not enabled. Please contact #sauron-support for help.”
To investigate if there were any configuration changes, we can utilize Grafana’s Swagger UI available at https://api.<pca><domain>/#/Grafana/getGrafanaConfig. This provides a convenient method to inspect all API endpoints and verify any modifications that might have occurred.
Swagger UI before Patch
The configuration section can be accessed at /v1/grafana/email/alerts/config.
Swagger UI after Patch
The mail template section contains various settings, there is still a config section, but it contains only a small amount of settings and old values are now shown.
Only values like smtpFrom, smtpHost, smtpUser and smtpPasswort can be set, but are not reflected.
cURL is your friend
Even in the absence of a the former Swagger section where I was able to upload the whole config, which seems to be a potential bug, we can attempt to retrieve the data using cURL. You can execute this command from a Management Node. If you’re running the command from an external workstation, ensure to include the –-insecure flag in the cURL command to bypass certificate issues. Make sure to set your Grafana admin password within the command.
[root@pcamn03 ~]# curl --location 'https://api.<pca>.<domain>/v1/grafana/config' --user 'admin:<password>' [auth.anonymous] enabled = false [paths] provisioning = /etc/grafana/provisioning plugins = /var/lib/grafana/plugins [auth.proxy] enabled = false auto_sign_up = false sync_ttl = 10 [emails] templates_pattern = emails/*.html [users] allow_sign_up = false auto_assign_org = true auto_assign_org_role = Viewer [alerting] enabled = true [unified_alerting] enabled = false
Hey, where did my SMTP section go? Are there other changes? It appears that both the [smtp] and [server] sections have been eliminated, and a new section labeled [email] has been introduced.
Let’s fix it!
Once more, cURL is your friend. Upload the configuration anew, and you’ll be prompted to enter the admin user password. The –data-binary parameter contains the configuration. Take care about the characters.
curl -u admin -k -X PUT "https://api.<pca>.<domain>/v1/grafana/config" \ -H "accept: application/json" \ -H "Content-Type: application/x-yaml" \ --data-binary @- <<EOF [auth.anonymous] enabled = false [paths] provisioning = /etc/grafana/provisioning plugins = /var/lib/grafana/plugins [server] enable_gzip = true domain = <pca>.<domain> root_url = https://grafana.<pca>.<domain> [auth.proxy] enabled = false auto_sign_up = false sync_ttl = 10 [users] allow_sign_up = false auto_assign_org = true auto_assign_org_role = Viewer [smtp] enabled = true host = <mailserver>:<port> skip_verify = true from_address = <sender-mail> from_name = <sender-name> EOF
After the configuration change, Grafana restarts automatically and is not reachable for a couple of seconds.
Back in the Game
Let’s initiate a test send in Grafana’s Alert section again – and there we have it: the email was sent successfully. Check your inbox too!
